Skip to content

test: measure first-attempt workflow conformance - #142

Merged
trvon merged 1 commit into
feat/lifecycle-inspectionfrom
test/workflow-first-attempt-metrics
Oct 6, 2026
Merged

trvon merged 1 commit into
feat/lifecycle-inspectionfrom
test/workflow-first-attempt-metrics

Conversation

@trvon

@trvon trvon commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Stack

Depends on #140; targets feat/lifecycle-inspection. Review after the testing-contract and lifecycle layers. No merge or auto-merge requested.

Why this layer

The historical routing evaluation documented repaired workflow-schema attempts despite final routing success. The current unchanged-copy baseline did not reproduce that problem: one requested model/configuration, six existing fixtures, all four checklist items passing and zero repairs. Compact tool copy, validator, and scenario prompts remain unchanged; this PR does not claim a guidance fix or model-performance improvement.

Changes

  • Report firstAttemptValid and firstAttemptSemantic separately from the existing four routing judgments. Repaired first calls stay first-pass failures; accepted-but-wrong arguments remain distinct from schema validity.
  • Preserve critical-success rules, accuracy, retries, timing, competing-controller rejection, and the full three-task first batch.
  • Extract a pure evaluator and add deterministic expectations/non-mutation controls. Enable discovery and lint for the new MJS test/helper and existing affected harness.
  • Capture bounded response provider/model/API metadata separately from the requested model string, plus Pi/Node versions. Unknown metadata stays unknown; omitted observations are counted.
  • Validate the existing published workflow example with the production parser and LoopStore; test cadence/rework misuse and cadence's inability to choose an outcome.
  • Document measurement boundaries, the bounded current baseline, and the evidence required before future copy tuning.

Copilot AI lite review requested due to automatic review settings October 6, 2026 19:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Two moderate evaluator issues remain around blank API metadata and first-attempt batch detection.

Review effort: Lite
Findings: None

What changed in this PR

Adds first-attempt workflow-conformance measurements, deterministic evaluator tests, metadata collection, and documentation while preserving existing routing behavior.

Changes:

  • Separates first-attempt validity from semantic correctness.
  • Adds workflow, evaluator, and cadence/rework tests.
  • Expands MJS discovery/linting and documents measurement limitations.
File Summary
vitest.config.ts Discovers TypeScript and MJS tests.
test/​workflow-guidance.test.ts Validates workflow examples and cadence semantics.
test/​e2e/​controller-routing-evaluation.mjs Provides evaluation and metadata helpers; changes requested for blank API handling and first-batch detection.
test/​e2e/​controller-routing-conformance.mjs Integrates evaluation and runtime metadata reporting.
test/​controller-routing-evaluation.test.mjs Adds deterministic evaluator and non-mutation tests.
docs/​CONTROLLER_ROUTING_EVAL.md Documents measurement boundaries and baseline limitations.
biome.json Enables linting for new MJS files.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@trvon
trvon added this pull request to stack #144 October 6, 2026 20:14
@trvon
trvon merged commit 10bf730 into feat/lifecycle-inspection Oct 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants